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(54) Automatically producing an image of a portion of a photographic image 



(57) A method of producing an image of at least a 
portion of a digital image, the digital image comprising 
pixels, comprising the steps of: 

a) computing a belief map of the digital image, by 
using the pixels of the digital image to determine a 
series of features, and using such features to assign 
the probability of the location of a main subject of 
the digital image in the belief map; 

b) determining a crop window having a shape and 
a zoom factor, the shape and zoom factor determin- 
ing a size of the crop window; and 

c) cropping the digital image to include a portion of 
the image of high subject content in response to the 
belief map and the crop window. 
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9a "’™' ” P '° a ”" 9 "" imaSa 01 3 POni " ‘ ph ‘ ,t ° 9ra <’ hl0 "™S* b, using digital 

[00021 For many decades, Iradnional ctemeecial photoflnisning systems have plaeed Units on the features offered 
to consumers to promote mass production. Among those features that are unavailable conventionally zooming and 
cropping have been identified by both consumers and photofinishers as extremely useful additional features thatLuld 
potential^ improve the quality of the finished photographs and the subsequent picture sharing experiences With the 

9i “' ™ 9in9 ' ^ ““ *> '">«•"»' Pbo'dOraphy 

H . ybnd ® nd digital P hoto 9 ra phy provide the ability to crop undesirable content from a picture, and magnify or 

zoom^Lh 6 ^ C °H l ° thS ent ' re photographic print ln s P ite of «he fact that some traditional cameras with 
n^tonr?nh ty prov,da con ® umerc 9 reater control over composing the desired scene content, studies have found that 

22J2SET f W ' Pe 3 C6r1ain am ° Unt ° f Cropping and zoomin 9 when viewing the finished photo- 
Kw £ t lmpreClse v,ewfinders of many point-and-shoot cameras, as well as simply second-guessing their 
initial compositions, are factors in the desirability of zoom and crop. In addition, it may be desirable to use some other 

Z^f ,1 P 'ri SU ?' “ ° Vate ' hsm *>“"*. I" anofh., seen.™, «om. “e““e Z m t" 

bdun £ “of^biS? t0n<l “ Pe ' , ° m ’ m ° re ag9re “ IV6 « scrapbook, e g , cutting along lire 

1 ™ M] T ^ re ara signi,icant differences in objectives and behaviors between these two types of cropping namely 
scribTriTf 19 a °rf SCrapbook making ’ Wlth ,he latter more difficult to understand and summarize. The invention de^ 

a lT tC Z T in9 and Cr0PPin9 f0r makin9 Photographic prints. One customer focus group 
study indicated that it would be beneficial to provide customers a double set of prints - one regular and one zoom 

rbout 0 hoithe 7 P r6d *T the Cr ° PP,n9 and Z ° 0min9 be d ° ne automatic ally. Most customers do not want to think 
and plc^res^ acc^pt^ls 9 * beln9 done 86 lon9 35 ,fld con,en ' and quality (e.g., sharpness) of tbe cropped 

S t ;i e l h n l Sb J?: litt ' S r6SearCh ° n aut0maticz °om and crop due to the apparent difficulty involved in performing 
such a task. None of the known conventional image manipulation software uses scene content in determining the 
automatic crop amount. For example, a program entitled "XV, a freeware package developed by John^adley at 

for 'I!!'n ? f ennsy vama ’^ JSA (Department of Computer and Information Science), provides an "autocrop" function 
for manipulating images and operates in the following way: H 

right P s°des m eX0m ' neS 3 b ° rder line ° f an image - in al1 of the four directions, namely from the top, bottom, left and 
the program checks the variation within the line. In grayscale images, a line has to be uniform to be cropped In 

of Sxer?oTth^l^\ SPat ' a T°L r ? latl0n and SPeCtra ' correlation have t0 be low - except for a small percentage 
amount ol variation^ 9 Cr ° PP ' n9 ' ° ther words ' a line wiN not be crapped if it contains a significant 

if a Hne along one dimension passes the criterion, the next line (row or column) inward is then examined; and 
the final cropped image is determined when the above recursive process stops. 

ZTLl h,S Pr ° 9 Z m eSSeatially tries t0 remove relatively homogeneous margins around the borders of an image It 

borde roZ TL C ° nt6nt ° f *" i™ 9 *- PraCtiC6 ' ,hS XV program is ■" cropping out the dark 

border generated due to imprecise alignment during the scanning process. However, disastrous results can often be 

“not 6 aPP r m i' aCk ° f SCene Unde(Sta " di " g - extreme cases, the entire image can be cmp ed 
[0007] Another conventional system, described by Bollman et al. in U.S. Patent 5,978,51 9 provides a method for 
cropping images based upon the different intensity levels within the image. With this system, an image to be cropped 
»"■> «««<■ «° bon-overtapping Mucks. Tba maab and ,a Jnc o, inZCSZS 
aii hi J ea ° b ° Ck ' BaS6d ° n the dlstnbut, °n of variances in the blocks, a threshold is selected for the variance 

a L ief C mnn a d t na T if erthan thethreshold variance are selected as regions of interest. The regions of interest 
are then cropped to a bounding rectangle. However, such a system is only effective when uncropped images contain 
regions where intensity levels are uniform and other regions where intensity levels vary considerably The effectiveness 
of such a system is expected to be comparable to that of the XV program. The difference is that the XV program 

=T? 9 7 * " ne faSh '° n *° id6ntify Unif0rm areas ’ While Bollman examines the image in a block 

by block fashion to identify uniform areas. 

mnnfi ,n summar y’ botb techniques cannot deal with images with nonuniform background 

pV onnn ? ,he ear1ier inventio " disclosed in U.S. Patent Application Serial No. 09/490,915, filed January 
25, 2000, the zoom factor needs to be specified by the user. There is, therefore, a need for automatically determining 
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the zoom factor in order to automate the entire zoom and crop process. 

[0010] Some optical printing systems have the capability of changing the optical magnification of the relay lens used 
in the photographic copying process. In U.S. Patent 5,995,201 , Sakaguchi describes a method of varying the effective 
magnification of prints made from film originals utilizing a fixed optical lens instead of zoom lens. In U.S. Patent 

5 5,872,61 9, Stephenson et al. describe a method of printing photographs from a processed photographic filmstrip having 

images of different widths measured longitudinally of the filmstrip and having heights measured transversely of the 
filmstrip. This method uses a photographic printer having a zoom lens and a printing mask to provide printed images 
having a selected print width and a selected print height. In U.S. Patent 4,809,064, Amos et al. describe an apparatus 
for printing a selected region of a photographic negative onto a photosensitive paper to form an enlarged and cropped 

io photographic print. This apparatus includes means for projecting the photographic negative onto first and second zoom 
lenses, each of the zoom lenses having an adjustable magnification. In U.S. Patent 5,872,643, Maeda et al. describe 
a film reproducing apparatus that can effectively perform zoom and crop. This apparatus includes an image pick-up 
device which picks up a film frame image recorded on a film to generate image data, an information reader which reads 
information about photographing conditions of the film frame image, and a reproducing area designator which desig- 
ns nates a reproducing area of the film frame image. However, the reproducing area of the film frame image is determined 
based on pre-recorded information about the position of the main object, as indicated by which zone of the photograph 
the automatic focusing (AF) operation in the camera was on - part of the recorded information about photographing 
conditions. In all the above-mentioned optical printing systems, the position of the photographic film sample and mag- 
nification factor of the relay lens are pre-selected. 

20 [0011] According to the present invention, there is provided a solution to the problems of the prior art. It is an object 

of the present invention to provide a method for producing a portion of a photographic image by identifying the main 
subject of the photographic image. 

[0012] According to a feature of the present invention, there is provided a method of producing an image of at least 
a portion of a digital image, comprising the steps of: 

25 

a) providing a digital image having pixels; 

b) computing a belief map of the digital image, by using the pixels of the digital image to determine a series of 
features, and using such features to assign the probability of the location of a main subject of the digital image in 
the belief map; 

30 c) determining a crop window having a shape and a zoom factor, the shape and zoom factor determining a size 

of the crop window; and 

d) cropping the digital image to include a portion of the image of high subject content in response to the belief map 
and the crop window. 

35 [001 3] One advantage of the invention lies in the ability to automatically crop and zoom photographic images based 

upon the scene contents. The digital image processing steps employed by the present invention includes a step of 
identifying the main subject within the digital image. The present invention uses the identified main subject of the digital 
image to automatically zoom and crop the image. Therefore, the present invention produces high-quality zoomed or 
cropped images automatically, regardless whether the background is uniform or not. 

40 [001 4] The foregoing and other objects, aspects and advantages will be better understood from the following detailed 

description of a preferred embodiment of the invention with reference to the drawings, in which: 

Fig. 1 is a schematic diagram of a system embodiment of the invention; 

Fig. 2 is a schematic architectural diagram of an embodiment of the invention; 

45 Fig. 3 is a schematic architectural diagram of an embodiment of the invention; 

Fig. 4 is a schematic architectural diagram of an embodiment of the invention; 

Fig. 5 illustrates the application of the invention to a simulated photograph; 

Fig. 6 illustrates the application of the invention to a simulated photograph; 

Fig. 7 illustrates the application of the invention to a simulated photograph; 

30 Fig. 8 illustrates the application of the invention to a simulated photograph; 

Fig. 9 illustrates the application of the invention to a simulated photograph; 

Fig. 10 illustrates the application of the invention to a simulated photograph; 

Fig. 11 illustrates the application of the invention to a simulated photograph; 

Fig. 12 illustrates the application of the invention to a simulated photograph; 

55 Fig. 13 is an exemplary uncropped photograph; 

Fig. 14 is a belief map of the image shown in FIG. 13; 

Fig. 15 is a cropped version of the image shown in FIG. 13; 

Fig. 17 is a belief map of the image shown in FIG. 16; and Fig. 18 is a cropped version of the image shown in FIG. 16. 
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tL! 5 Th o mvent '° n automatlcal| y zooms and crops digital images according to an analysis of the main subject in 

tvoe Dhotea^h 10118 ^ 3 * ySten ! f0r deteCting main subjects ( e -3 . m ain subject detection or »MSD“) in a consumer- 
S® P . h ° g aP f hlC '^ aga [; om the Perspective of a third-party observer has been developed and is described in U S 

' referenc^M? hl^ 0 ' 09/223 ' 860 ' flled December 31,1 998, the disclosure of which is incorporated herein by 
IlZ H r? , ?' 111 " Pr0Vid6S 8 meaSUre ° f Saliency or re,ative importance for differed regions that are 
content w r k ' SUdJ6CtS m an ,mage - Main subject detection enables a discriminative treatment of the scene 
rooifii Conventinff f appllcatl0n ® r ® ,ated to consumer photographic images, including automatic crop and zoom 
Lrh J ^ onvent onal Wlsd ° m ln the field of computer vision, which reflects how a human observer would perform 
, conrenr ma r SUbjSCt detectlon and cropping, calls for a problem-solving path via object recognition and scene 
nt determination according to the semantic meaning of recognized objects. However generic obiect recoonition 

~ ™ P" 6 *" academia and indu.J 9 ' SC09n “°" 

maHahia 1 imk^cn 1,Ste . n '"u" 11 * upon mos *V low-level vision features wwi semantic information integrated whenever 

“““ ST a * "” bsr 01 S “ Wa8kS ' '"*** ^"."Mon, perceptual grouping Teamm 

extraction; and probabilistic and semantic reasoning. In particular, a large number of features are extractor earh 

mnlfll 9 '^ ' n th h IT 9 ® *° represent a wide variet V of visual saliency properties, which are then input into a 
rams ?f enS I b a! n Pr0 rty netWOrk t0 9 enerate a belief map containing a continuum of values. 

in the imanT™^ 0 ' r f s ' ons that belon 9 to the main subject are generally differentiated from the background clutter 
the image. Thus automatic zoom and crop becomes possible. Automatic zoom and crop is a nontrivial operation 
that was considered impossible for unconstrained images, which do not necessarily contain uniform background with 

rirr oum i scene understandin 9 - si; 

S ° n r ply USing a Centered cr0p at a fixed zoom (magnification) facto" or removing the untrmback 
?O019l d 7 71 bontow - Th ® centered crop has been f °und unappealing to customers 
Ske ihLd for hS t t US f d by the inVenti ° n iS 3 Mst 0f se 9mented regions ranked in descending order of their 
ntn a mf h 2 ff ff mam SUbjSCtS f ° r 3 generic or Specific a PP»c a tion. This list can be readily converted 
mlnrnT TT thebnghtness of a re 9 io " is proportional to the main subject belief of the region Therefore this 

mapcan be called a mam subject "belief map. This "belief map is more than a binary map that only indicates iocation 
of he determined main subject. The associated likelihood is also attached to each region so tha/regions with lame 
rnnom co * Tespond t0 re 9'°ns with high confidence or belief of being part of the main subject 9 

hPranL I°« S ° m ? 6 f ent ' th ' S bel ' ef map reflects the inherent uncertainty for humans to perform such a task as MSD 
because different observers may disagree on certain subject matter while agreeing on other subject matter in terms 

° ” bj r H T Ven 3 bina,V deCiSi ° n ’ When deSired ' ca " be opined by 

ZTJ h ? P f MOre ° Ver ' the belief information ma V be useful for downstream wlkS^SST 
?00211 ^ be aS t !! 9ned t0 diff6rent re9i ° nS (SUbject matters > in determining the amount of crop ' 

m2 min tn a de * e ™ t,on of crop - lnvention uses the main subject belief map instead of a binarized version of 
ap il® . '? 7 7 3 1 Cropplng decision that is 'Reversible. Furthermore, using the continuous values of the 
“ J0Ct 7' efS halps trade -° ff different regions under the constraints encountered in cropping. A binary dec^Sn 

reo^nt 2m 1 fh 3nC lr at t0 mC ' Ude ’ 0306 made ' leaves little room for tra de-off. For example 9 if the main subject 
9 * 7 ef • h3n ® crop window ’ the on| y reasonable choice, given a binary main subject map is to leave equal 

amounts of margin around the main subject region. °n the other hand, secondly main subjects arelndicItedTv fowe 
belref values ,n the mam subject belief map, and can be included according to a descending orde! bet^valuI oZ 

excluders made be " ef ValU6S are inclUdS(1 Moreover ’ if an undesirable binary decision on what to include/ 

thL thr! h Tf I f If 00 r6C0UrSe t0 °° rreCt the mistake ' Consequently, the cropping result becomes sensitive to 
77 h ° d US f d j° ° btain the b ' nary decision - With a continuous-valued main subject belief map every region or 
mnool S ass °7 ted Wltb a llkellbood of being included or a belief v a| ue in its being included. ’ ^ 9 

[ ] To reduce the degrees of freedom in determining the amount of crop, and to limit the amount of resolution 

~ " part ' cu ' ar f° r mak ' n 9 Photographic prills, in one 

l?7f! l h 7f J-,f L b Z00m fact0rs t0 the ran 9 e of [1 2, 4], This is based on the findings in the customer focus 

[00231 To reduce mJd 3ft that the present invention could be used with any the zoom factor 

[ 023] To reduce the degrees of freedom in determining the amount of crop, in particular for makina ohotoaraohir 

bid :r;r dim i the invention restdcts the set ° f aii ° wabie — ^ 

based on the findings in the customer focus studies. In addition, an extremely large zoom factor usua hr leads to blu™ 

If hire 6P ? b ' e P « "I t0 the ' imit imP ° Sed bV the reSOlution of the 'mage. Ja zoom^ ^flor Stalled 

in rH 'T ' 0n fa S W,thin the ran9e ° f acceptable zoom factors (e.g., between 1 .2 and 4.0) it will be used 
end subsequent cropping process. Otherwise, the zoom factor is clipped to 1 .2 at the lower end and 4.0 at the higher 

re° 241 h K ef T 9 10 , F ' 9 ' ’’ the followin 9 description relates to a digital printing system. A source diqital image 10 is 

ZTt I" im39e Pr ° CeSSOr 2 °- The di9ital image processor 20 may be conne^eTtl JlneTa, con ^ 

computer 40 under operator control from an input control device 60. The monitor device 50 displays diagnostic infor- 
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mation about the digital printing system. The general digital image processor 20 performs the needed image processing 
to produce a cropped and zoomed digital image 99. 

[0025] Referring to Fig. 1 a, the following description relates to an optical printing system. A photographic film sample 
31 is received by a film scanner 32 which produces a source digital image 1 0 relating to the spatial density distribution 
5 of the photographic film sample. This source digital image is received by a digital image processor20. The digital, image 
processor 20 may be connected to a general control computer 40 under operator control from an input control device 
60. The monitor device 50 displays diagnostic information about the optical printing system. The general control com- 
puter 40 keeps track of the lens magnification setting. 

[0026] Referring to Fig. 2, a zoom factor 11 , which corresponds to the lens magnification setting may also received 
io by the image processor 20 from the general control computer 40 under operator control. The image processor 20 

receives the source digital image 10 and uses the zoom factor 11 and the source digital image 10 to calculate the 
proper position for the photographic film sample in the form of a film sample position 9. The photographic film sample 
is positioned in a gate device 36 which holds the film negative in place during the exposure. The gate device 36 receives 
the film sample position 9 to position the photographic film sample to adjust which portion of the imaging area of the 
is photograph will be printed. 

[0027] Referring to Fig. la, a lamp house 34 provides the illumination source which is transmitted through the pho- 
tographic film sample 31 and focused by a lens 12 onto photographic paper 38. The time integration device 13 opens 
and closes a shutter for a variable length of time allowing the focused light from the lamp house 34 to expose the 
photographic paper 38. The exposure control device 16 receives a brightness balance value from the digital image 
20 processor 20. The exposure control device 16 uses the brightness balance value to regulate the length of time the 

shutter of the time integration device stays open. 

[0028] A block diagram of the inventive cropping process (e.g., the digital image understanding technology) is shown 
in Fig. 3, which is discussed in relation to Figs. 5-12. Figs. 5-12 illustrate the inventive process being applied to an 
original image shown in Fig. 5. 

25 [0029] In item 200, the belief map is created using MSD. The present invention automatically determines a zoom 

factor (e.g. 1 .5X) and a crop window 80 (as shown in Fig. 7), as referred to in item 201 of Fig. 3. This zoom factor is 
selected by an automatic method based directly on the main subject belief map (e.g., an estimate of the size of the 
main subject). The crop window is typically a rectangular window with a certain aspect ratio. After the zoom factor is 
determined by the digital image processor 20, the value of the zoom factor is used subsequently by the digital image 
30 processor 20 shown in Fig. 1 . In Fig. la, the zoom factor is used to communicate with the lens 12 to adjust the. lens 
magnification setting. This adjustment allows the lens 1 2 to image the appropriate size of the photographic film sample 
31 onto the photographic paper 38. 

[0030] In item 201 , regions of the belief map are clustered and the lowest belief cluster (e.g., the background belief) 
is set to zero using a predefined threshold. As discussed in greater detail below, sections of the image having a belief 
35 value below a certain threshold are considered background sections. In item 202 such sections are given a belief of 
zero for purposes of this embodiment of the invention. 

[0031] Then, in item 202 the centroid, or center-of-mass (used interchangeably hereon forth), of nonzero beliefs are 
computed. More specifically, in Fig. 5 the subject having the highest belief in the belief map is the woman and the 
stroller. Fig. 7 illustrates that the centroid of this subject is approximately the top of the baby's head. 

^0 [0032] The centroid ( x , y) of a belief map is calculated using the following procedure: 



x = '£x l b el(x l ,y.), y = 

i i 

45 , 

where x,and y i denote that coordinates of a pixel in the belief map and bel(x f ,y$ represents the belief value at 
this pixel location. 

[0033] Before the crop window is placed, a proper crop window is determined in item 203. Referring to Fig. 4, there 
so js shown a block diagram of a method that automatically determines a zoom factor in response to the belief map. In 
item 301 , two second-order central moments, and Cyy, with respect to the center-of-mass, are computed using the 
following procedure: 
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c = 



-■*)* xbel (x i ,y i ) Y j (y i - y ) 2 x be! (x,,y t ) 



S be Kx n yi) 



C — - / 
9 yy 



E bel ( x ,>y.) 
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[0034] Note that these two terms are not the conventional central moments that are computed without any weiqhtinq 
unctj ons ,,ln the preferred embodiment, a linear weighting function of the belief values is used. However, the conven 

rnn-fei 6 a 13 «° mentS ' ° r Centfal moments b V a nonlinear function of the belief values, can also be used 

[0035 An effective bounding rectangle (MBR) of the regions of high subject content can be calculated usinq the 

following procedure, where the dimensions of the MBR are calculated by calculated using the 
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hMri 6 L/inn 6 '" USt ? t , eS th3t thS effective b °unding rectangle 70 is centered at approximately the top of the boy’s 
head and approximately encompasses the region of high subject content. In general, the aspect ratio of the oriqinal 

thrSriain^ a ^ tain t ed 't Th t e hT re ' 3 Cr ° P W ' nd0W 80 ' S detemined in rtem 303 stJ ch that it is the smallest rectangte of 
the original aspect ratio that encompasses the effective MBR 70. y 

S I? 6 " 2 ° 4 ’ the initia ' position of the cr °P window p 80 is centered at the centroid, as shown in Fiq 7 
[0036] The crop window is 80 then moved so that the entire crop window is within the original image (e g item 205) 
as shown in F,g. 8. In item 206, the crop window 80 is moved again so that all the regions of the highest belief v^ues 

i "SEST a th “ T * the Cr ° P Wind ° W and *° Cfeate 3 mar ^ in 81 • - s'°wn in FIG. ^This JoLsI (e 
g. , 6) captures the entire subject of interest. Therefore, as shown in Fig. 9, the top of the woman's head is included 

m the crop window^ Compare this to Fig. 8 where the top of the woman's head was’outside the crop w"dow 

[0039] Decision box 207 determines whether an acceptable solution has been found, i.e., whether it is possible to 

include at least the regions of the highest belief values in the crop window possible to 

[0040] If an acceptable solution exists, the window is again moved, as shown in item 208, to optimize a subject 
content index for the crop window. The preferred embodiment of the present invention defines the subject content 

numerical beliT ° ^ Va ' UeS 'T*' Cr ° P window ' 11 should be noted that the present invention specifies higher 
numerical belief values corresponding to higher main subject probability. Therefore, finding a numerical maximum of 

whemTh ! V k U6S ' S ® qUIValent t0 findin 9 an °P timu m of the subject content index. This is shown in Fig 1 0 

Turn of bel^ fH? afy ° bjeCts (e ' 9 ; flowers > are inc| uded within the crop window 80 to increase the sum of beliefs The 
sum of beliefs for a crop window is computed as follows. 



40 



45 



sum(w) = £ bel(x,y), 

(*,>»)€» 

mn "bp 6 be }*' % re P resents the belief value at a given pixel location (x, y) within the crop window w 
[0041] Provided that the primary subjects are included, moving the crop window so that more of the secondary 
subjects are included would increase the sum of belief values within the crop window. Recall that the primary subjects 
are in icated by the highest belief values and the secondary subjects are indicated by belief values lower than those 

th? h T m t arV subjects but hl 9 her than those of the background subjects. The goal is to find the crop window that has 
the highest sum of belief values while ensuring that the primary subjects are completely included in the crop window i e 



50 



55 



w= max sum (w), 

H *LW 

where W denotes the set of all possible crop windows that satisfy all the aforementioned constraints (e q those 

mn*^ e C ° mp ' etely wrthm the uncr °PP ed image and those that encompass the entire primary subjects) 

21 , 2 ° n P ' aCe °' rtem 2 ° 9, n0t Sh ° Wn) ' the position of the center of the cr °P window is used to 
calculate he translational component of the film sample position 9. The gate device 36, shown in Fig 1 a receives the 

r,r; e ^ Slt ' 0n ?r H dUS ! S thlS ,nformation t0 control to* Ct the Photographic film sample 31 relative to 

the lens 12. Those skilled ,n the art will recognize that either or both of the lens 12 and the photographic film sample 
31 may be moved to achieve the centering of the effective cropped image region on the photographic paper 38 
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[0043] Referring to Fig. 3, if decision box 207 does not produce an acceptable solution, the final position of the crop 
window is restored to that of item 205. Then, referring to Fig. 1 a, the position of the center of the crop window is used 
to calculate the translational component of the film sample position 9. The gate device 36, shown in Fig. 1 , receives 
the film sample position 9 and uses this information to control the position of the photographic film sample 31 relative 
5 to the lens 12. 

[0044] The simulated image example shown in Figs. 5-12 illustrates the progress the invention makes as it moves 
through the process shown in Fig. 3. One could formulate the problem as a global exhaustive search for the best 
solution. The procedure used in the invention is considered a ‘'greedy*' searching approach and is certainly more efficient 
than conventional processes. 

io [0045] The invention utilizes a built-in "k-means" clustering process to determine proper thresholds of MSD beliefs 
for each application. The invention also uses clustering, as discussed below to enhance the cropping process. In one 
preferred embodiment, it is sufficient to use three levels to quantize MSD beliefs, namely “high", "medium", and “low." 
As would be known by one ordinarily skilled in the art, the invention is not limited to simply three levels of classification, 
but instead can utilize a reasonable number of classification levels to reduce the (unnecessary) variation in the belief 
is map. These three levels allow for the main subject (high), the background (low), and an intermediate level (medium) 
to capture secondary subjects, or uncertainty, or salient regions of background. Therefore, the invention can perform 
a k-means clustering with k = 3 on the MSD belief map to “quantize" the beliefs. Consequently, the belief for each 
region is replaced by the mean belief of the cluster in that region. Note that a k-means clustering with k = 2 essentially 
produces a binary map with two clusters, "high" and "low," which is undesirable for cropping based on earlier discussion. 
20 [0046] There are two major advantages in performing such clustering or quantization. First, clustering helps back- 

ground separation by grouping low-belief background regions together to form a uniformly low-belief (e.g., zero belief) 
background region. Second, clustering helps remove noise in belief ordering by grouping similar belief levels together. 
The centroiding operation does not need such quantization (nor should it be affected by the quantization). The main 
purpose of the quantization used here is to provide a threshold for the background. 

25 [0047] The k-means clustering effectively performs a multi-level thresholding operation to the belief map. After clus- 

tering, two thresholds can be determined as follows: 

threshold tow HC fow +C m ^ ium )i 2, 

30 

threshold hfgh = (C medium + C hjgh )l2 

where {C toW) C medjum} C hfgf j is the set of centroids (average belief values) for the three clusters, and threshold hw 
35 and threshold hjgh are the low and high thresholds, respectively. 

[0048] Regions with belief values below the lower threshold are considered "background" and their belief values are 
set to zero in items 202, 302 and 402 discussed above. Regions with belief values above the higher threshold are 
considered part of the main subject and need to be included in their entirety, whenever possible. Regions with inter- 
mediate belief values (e.g., less than or equal to the higher threshold and greater than or equal to the lower threshold) 
40 are considered part of the "secondary subject" and will be included as a whole or partially, if possible, to maximize the 
sum of main subject belief values retained by the crop window. Note that the variance statistics on the three clusters 
can be used to set the thresholds more accurately to reflect cluster dispersions. 

[0049] The invention initializes the k-means process by finding the maximum value be/ max and minimunrvvalues 
be/ mjn j mum of the belief map, computing the average value bel average of the maximum and minimum values for item in 
45 the belief map, and setting the initial centroids (denoted by a superscript of 0) at these three values, i.e., - 

^low — ^^min imium * P medium ~ medium* ^ high ~ ^^max imum 

so [0050] Other ways of initialization may apply. For more about the 

k-means process, see Sonka, Hlavac, and Boyle, Image Procesing Analysis, and MachineVision, PWS Publish- 
ing, 1999 pagse 307-308. For typical MSD belief maps, the k-means process usually converges in fewer than 10 
iterations. 

[0051] In applications where a zoom version of the cropped area is desired, there are two scenarios to consider. 
55 First, the zoom version effectively requires higher spatial resolution than the highest resolution of the original data. 
However, a visible loss of image sharpness is likely of concern in the situation. Second, the zoom version effectively 
requires lower spatial resolution than the highest resolution of the original data. In both cases, the invention uses an 
interpolation process to resample the data in order to retain a maximum amount of image detail. In general, edge or 
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rzll mt0 numerica ' values for calculat ' n 9 the sum of the belief values discussed above Finally Figs 15 and e 
illustrate images cropped according to the invention. y ’ gs ' 1 ° and 1 8 

E ] ahH°t r th6 !'° USe " PiCtUre ' b ° th Bradley and Bollman < u s - Pa tent 5,978,519) would keep the entire imaae and 

- br.^L P ^S , r.l« l0 '' e ' ,a ”" P °”° n ,he P ““' e ” a k “» “» 

22 «£ | n he in t V6nti0n been described ^ ‘erms of preferred embodiments, those skilled in the art will rec 
2SJ Th TT* Can practiced With modi,ication within the spirit and scope of the appended claims 



Claims 



35 



40 



45 



50 



55 



1 . A method of producing an image of at least a portion of a digital image, the digital image comprising the steps of: 

a) providing a digital image having pixels: 

of fir: 9 a H belief maP „° f the di9ital image ’ by using the pixels of the di gKal i™»ge to determine a series 
image 1 ki^he^ief S map* UCh * 388,90 ** Pr ° babi ' ity ° f the l0Cati °" ° f 3 main «**« of the digital 

i *. “ircr , ”' in9 a shape ma a zoom me shaw » a • 
irsr^ 9 ssr 101 inc " ,api ai ^ ,hei 

2. The method of claim 1 wherein step c) further comprises the steps of: 

i) computing a weighted center-of-mass of the belief map, weighted by the belief values of the belief mao 

"> co "»“ ll "9 »«9b«b ‘WM ~te of the belief map, relative toll* cm JoPmSs a £ m SZZ . 
weighting function of each belief value of the belief map; d weighted by a 

iii) computing an effective rectangular bounding box according to the central moments; and 

at. m ,r e ™p a v.2w h “'"’ 9 8 Shape and “ Z “ m ,ac,0, ■ ,h8 SBape aM “ om • 

3. The method of claim 1 wherein step d) further comprises the steps of: 

i) selecting an initial position of the crop window at a location which includes the center of mass; 
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ii) using the belief values corresponding to the crop window to select the position of the crop window to include 
a portion of the image of high subject content in response to the belief map; and 

iii) cropping the digital image according to the position of the crop window. 

5 4. The method of claim 2 wherein step d) further comprises the steps of: 

i) selecting a crop window of a rectangular shape and of an identical aspect ratio to the (uncropped) digital 
image; and 

ii) selecting a zoom factor to determine the size of the crop window such that it encompasses the effective 

io bounding box. 

5. The method of claim 2 wherein the weighting function in step b) is a linear weighting function. 

6. The method of claim 2 wherein the weighting function in step b) is a constant function. 

15 

7. The method of claim 3 wherein step b) further comprises the steps of: 

i) calculating a subject content index for the crop window derived from the belief values; 

ii) following a positioning procedure of repeating step i) for at least two positions of the crop window; and 

20 Iii) using the subject content index values to select the crop window position. 

8. The method of claim 1 wherein the crop window is completely within the digital image. 

9. The method of claim 2 wherein step b) further comprises the step of performing a clustering of the belief map to 

25 identify at least a cluster of highest belief values corresponding to main subject, a cluster of intermediate belief 

values corresponding to secondary subjects, and a cluster of lowest belief values corresponding to the background; 

10. The method of claim 9 wherein said clustering includes setting said background portions to a zero belief value. 

30 ii. The method of claim 5 further comprising positioning said crop window such that the subject content index of said 
crop window is at an optimum. 

12. The method of claim 3 further comprising positioning said crop window such that said crop window includes all of 
said main subject cluster. 

35 

13. The method of claim 12 further comprising positioning said crop window to include a buffer around said main 
subject cluster. 

14. A computer storage product having at least one computer storage medium having instructions stored therein caus- 

40 ing one or more computers to perform the method of claim 1 . 

15. A method of producing an image of a portion of at least a portion of a photographic image onto a photographic 
receiver, comprising the steps of: 

45 a) receiving a digital image corresponding to the photographic image, the digital image comprising pixels; 

b) computing a belief map of the digital image, by using the pixels of the digital image to determine a series 
of features, and using such features to assign the probability of the location of a main subject of the digital 
image in the belief map; 

c) determining a crop window having a shape and a zoom factor, the shape and zoom factor determining a 

so size of the crop window; and 

. d) locating the relative optical position of a photographic image, a lens assembly, and a photographic receiver 
in response to the belief map and illuminating a portion of the photographic image of high subject content to 
produce an image of such portion onto the photographic receiver. 

55 16. The method of claim 15 wherein step c) further comprises the steps of: 

i) computing a weighted center-of-mass of the belief map, weighted by the belief values of the belief map; 

ii) computing weighted central moments of the belief map, relative to the center-of-mass and weighted by a 
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10 



15 



weighting function of each belief value of the belief map; 

iii) computing an effective rectangular bounding box according to the central moments- and 
lSt S p wSw”"*" ha,ln9 • 51,1 ** “ a * Wor dstetnrining , 

17. The method of claim 1 5 wherein step d) further comprises the steps of: 

i)’ selecting an initial position of the crop window at a location which includes the center of mass- 

J US “ 19 th ® bellef Values corres P on ding to the crop window to select the position of the crop window to include 
a portion of the image of high subject content in response to the belief map; and 
iii) cropping the digital image according to the position of the crop window. 

18. The method of claim 1 6 wherein step d) further comprises the steps of: 

5 - imag^and 8 ^ Wi " d ° W ° f 3 rectangular shape and of an identical aspect ratio to the (uncropped) digital 

' bound“g n boi ZO ° m f8Ct ° r l ° de * emiine the Si2e of the cr °P window such th «t » encompasses' the effective 



20 19. The method of claim 1 6 wherein the weighting function in step b) is a linear weighting function. 

20. The method of claim 1 6 wherein the weighting function in step b) is a constant function. 

2s 21 • The method of claim 1 7 wherein step b) further comprises the steps of: 

i) calculating a subject content index for the crop window derived from the belief values- 

°! r6peating St6P 0 f ° r 3t leaSt tW0 P° sitions of the crop window; and 
li) using the subject content index values to select the crop window position. 

3° 22. The method of claim 15 wherein the crop window is completely within the digital image. 

23 ‘ ^ e ,T th °, d ° f Claim 1 6 wherein ste P b ) further comprises the step of performing a clustering of the belief mao to 

vahiescoiTp 881 ° f h 'f eSt belief values corresponding to main subject, a cluster of intermediate belief 

35 corresponding to secondary subjects, and a cluster of lowest belief values corresponding to the background. 

24. The method of claim 23 wherein said clustering includes setting said background portions to a zero belief value. 
25 ‘ 2 d ZZ5Z2TJ a ?oXT prisi " 9 posi,ionin9 said crop window such that the subject ind - * 



40 



45 



rfLTtS suSl 7 IT* C ° mPrl “’ 3 P0S ™' >nina ““ ‘" >P SUOh “ "» »' M °» IM"— all 

27 26 '“" ter com P r 'sing positioning said crop window to Mod. » buffer around said main 



28 * ^ or ^ puter storage product having at least one computer storage medium having instructions stored therein 
ing one or more computers to perform the method of claim 1 . 
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